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3D VISUAL INTERFACE FOR REMOTE CONTROL DEVICES 
BACKGROUND OF THE INVENTION 



to the display of the status of the various remotely controlled devices. 

Remote control of devices has been possible for a long time. At first, remote control and 
monitoring were done mechanically. More recently, remote control has been typically 
accomplished with electronics. As the technology advanced and the cost of remote control 
devices declined, home and industrial automation via remotely controlled devices has become 
widely feasible. Examples of popular families of home automation electronics remote control 
devices include Safety I s ', HomeLink, Stanley, Magnavox, IBM Home Director, and X-10. Other 
types of control devices are used for controlling multimedia equipment. 

There are typically two types of devices in a remote controlled system; controllers and 
remote modules. Remote modules are connected to devices to be controlled. In a home system, 
remote modules are typically connected to such things as lights, appliances, sprinklers, alarms, 
and the furnace. Various types of controllers also exist. The user uses a controller to control the 
various remote modules in the system and thus control the device connected to it. The most 
basic controller has a set of switches that allow the user to control the state of various remote 
modules. Other types of controllers allow the user to control remote modules via telephone, 
computer, or via a portable wireless controller. 

With the exception of the computer controller, none of these controllers have a graphics 
display to indicate the status of the various devices being controlled. In the case of the computer 
controller, the display presented is typically a 2D plan view of the environment. Many people do 
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not relate well to 2D imagery. Additionally, display of the effects of one controlled device on 
each other are not shown. For instance, if lawn sprinklers and an exterior light were both on, the 
spray from the sprinklers should be illuminated. Interactions cannot be displayed with 2D 
imagery. 

A variety of methods have been used to display the status of remotely controlled devices. 

U.S. patent No. 5148154 to MacKay et al. granted September 15, 1992 describes a 
multidimensional user interface. 

U.S. patent No. 5452414 to Rosendahl et al. granted September 19, 1995 describes a 
method of rotating a three-dimensional icon to provide additional information about an object 
available within the computer. 

U.S. patent No. 5388197 to Rayner granted February 7, 1995 describes a video editing 
system operator interface for visualization and interactive control of video material. 

U.S. patent No. 6043818 to Nakano et al. granted March 28, 2000 describes a graphical 
user interface for displaying a menu image on a display. 

U.S. patent No. 5973694 to Steele et al. granted October 26, 1999 describes a 
communication method using icons, text, and audio. 

U.S. patent No. 5555354 to Strasnick and Tesler granted September 10, 1996 describes a 
method and apparatus for navigating within a three dimensional graphic display space and 
manipulating information found in it. 

U.S. patent No. 5276785 to Mackinlay et al. granted January 4, 1994 describes a method 
of changing the viewpoint within a computer-driven three-dimensional display. 

None of the prior art cited displays the status of remote controlled devices on a display 
using a 3D graphics nor does any display of the effects of one controlled device on another. 
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SUMMARY OF THE INVENTION 

The present invention is a method for displaying the status of remote controlled devices 
using 3D graphics. The object of the present invention is to provide for realistically visualizing 
the effects of control actions locally through the use of computer graphics. A 3D database is 
prepared showing the controlled devices in their environment. For each controlled device there 
is software that alters the scene properties according to the actions of the controller. For 
instance, if a light were the controlled device, it might be shown as white when illuminated and 
black when extinguished. Additionally, when illuminated, the illumination of nearby objects 
would be shown. The present invention is comprised of a database, a controller, software that 
relate controls to the database, and a means to realistically render the database whereby the 
effects of controls are displayed realistically in 3D imagery. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 illustrates one embodiment of the present invention with various controlled 
devices in their environment with all controlled devices in the "OFF" state. 

Figure 2 illustrates one embodiment of the present invention with the displayed imagery 
5 with an exterior light turned on. 

Figure 3 illustrates one embodiment of the present invention with the displayed imagery 
with an interior light turned on. 

Figure 4 illustrates one embodiment of the present invention with the displayed imagery 
with some sprinklers turned on. 
□ 10 Figure 5 illustrates one embodiment of the present invention with the displayed imagery 

with some sprinklers and an exterior light turned on. 

Figure 6 illustrates one embodiment of the present invention with the displayed imagery 
^ with a security camera pointed to the left. 

y. Figure 7 illustrates one embodiment of the present invention with the displayed imagery 

ry 

CO 15 with a security camera pointed to the right. 

01 

JJ Figure 8 illustrates the processing logic employed in one embodiment of the present 

invention. 
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DETAILED DESCRIPTION OF THE INVENTION 

The following terms and definitions will be of use in the subsequent disclosure. 
Database - a mathematical model of the world, typically comprising polygons, surface 
properties and textures, used as input for 3D graphics rendering. 
5 3D database - a mathematical model of the world in three dimensions. 

Lighting model - a mathematical model of the illumination pattern produced by a light 
source. 

Rendering - the action of converting graphics primitives into individual pixels for an 
image to be displayed. 

□ 10 Monitoring - determining by independent means if a command action has actually 

% 5 occurred. 

^ With reference to Figure 1, various controlled devices in their environment are illustrated, 

p In this example interior light 1, exterior light 2, sprinklers 3, 4, 5, 6, 7, and 8, and security 
camera 9 are all turned off. 

s 

nj 15 With reference to Figure 2, the displayed imagery with interior light 1 turned on is 

m 

O illustrated. The illumination pattern from interior light 1 illuminates portions the building interior 

O 

wall 10, floor 11, and nearby furniture 12 on the inside of the building. 

The illumination pattern from light sources such as the interior and exterior lights 
discussed herein is determined by the lighting model of the light source. Lighting models for 
20 light sources are well known. The amount of illumination for a given point on an object, such as 
the spray pattern discussed later, is a function of the lighting model and the proximity of that 
point on the object to the light source. Rendering is accomplished by the conventional graphics 
accelerator card under control of the application software. 
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With reference to Figure 3, the displayed imagery with exterior light 2 turned on is 
illustrated. The illumination pattern from exterior light 2 illuminates portions the building 
exterior walls 13, porch 14, and portions of the lawn 15. 

With reference to Figure 4, the displayed imagery with sprinklers 3, 4, and 6 turned on is 
illustrated. 

With reference to Figure 5, the displayed imagery with exterior light 2 and sprinklers 3, 
4, and 6 turned on is illustrated. In addition the scene elements illuminated in FIG. 3, appropriate 
portions of the spray pattern from sprinklers 3, 4, and 6 are illuminated illustrating the effect of 
one controlled device on another. 

With reference to Figure 6 and Figure 7 the displayed imagery for two positions of 
security camera 9 is illustrated. The orientation of the graphic representation of security camera 9 
shows the direction it is aimed. Dotted lines 16, if present, indicate security camera 9 is active 
and show the field-of-view. 

With reference to the Figure 8, the 3D database 801 in step 1 can be prepared using 
conventional modeling software such as MultiGen or 3D Studio Max. The modeler typically 
works from scaled drawings and photographs to produce the 3D database using the modeling 
software. 

With reference to step 802, control actions produce changes in the 3D database. For 
example, operation of a remote door opener will change the database to reflect whether the door 
is currently opened or closed. Database representations of various changes are prepared ahead of 
time for later use. 

With reference to step 803, the state of the command for each controlled device is 
determined by monitoring the output from the controller. This can be done electronically in the 
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case of keypad type controllers or in software for the computer controllers. Software in the 
computer receives the control state data. The control state changes may be preprogrammed or 
input by the user. Various means can be used to permit user interface to the system. Examples 
include a wide variety of human input devices such as keypads with dedicated buttons for each 
action of each controlled device, computer keyboards in conjunction with computer software, a 
computer mouse in conjunction with computer software, and a touchscreen display in 
conjunction with computer software. 

With reference to step 804, software translates the command state into database changes 
selecting database changes from among those prepared in Step 802. Scene illumination 
parameters or other rendering parameters may also be changed. A table look-up may be used to 
select the changes. 

With reference to step 805, an image can be rendered from the database and illumination 
parameters using a commonly available graphics accelerator card or image generator in 
conjunction with realtime software such as WorldToolKit, V-Tree, VEGA, OpenGVS, or 
Performer. 

With reference to step 806, monitoring the condition of the controlled device can be 
accomplished by various means. This provides more rigorous monitoring relative to simply 
monitoring the commands since commands, as in step 803, may not always have the intended 
effect due to a hardware or software failure. Selecting the most appropriate monitoring means 
depends on the device being monitored, the fidelity of the monitoring needed, accessibility to the 
controlled device, and budget. Some possibilities include sensing the current going to a 
controlled device, sensing the position of a controlled device with switches, encoders, 
potentiometers, or photocells as appropriate, and monitoring fluid flow with an electronic 
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flowmeter. If the monitored result differs from the command, the monitored result is rendered. 
For example, a burned out light may be commanded to be illuminated but monitoring of the 
actual lamp illumination may determine the light is not illuminated. 

With reference to step 807, selecting the viewpoint from a list or positioning the 
5 viewpoint interactively is done via keyboard or other computer input device to the real-time 
software described in step 805. 

While the invention has been described in substantial detail with what is presently 
considered to be the most practical and the presently preferred embodiment, it is to be 
understood that the invention is not limited by the embodiment described, but rather, the 

f=4 . ... 

■■== 10 invention is intended to include various modifications and the various equivalents that are 

»s 

£ covered by the spirit and scope of the appended claims. For instance, the present invention can 

U3 

p also be used to monitor the state of various household devices such as kitchen appliances, home 

Li, entertainment systems, air conditioners, and industrial devices such pumps, valves, motors, 

s 

H conveyers, and electrically activated doors. Additionally, the present invention can be used on 

ry 

fU 15 vehicles to monitor the status of vehicle systems including those on automobiles, trucks, 

01 

^ airplanes, trains, ships, submarines, and spacecraft. Therefore, it is understood that all such 
changes, modifications, alterations, and equivalents are included within the scope of the 
following claims. 
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